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EXECUTIVE  SUMMARY 


A.  BACKGROUND/PROBLEM.  The  Army  is  currently  moving  a  significant  number 
of  sophisticated  weapon  systems  from  R&D  to  production.  This  transition 
process  is  bringing  to  light  a  considerable  amount  of  production  cost 
growth.  Initial  production  uncertainties  need  to  be  analyzed  and  con¬ 
tingencies  addressed  to  avoid  or  to  minimize  program  disruptions.  An 
approach  similar  to  the  one  used  to  address  R&D  uncertainties,  TRACE, 
appears  feasible  to  analyze  the  early  production  risks  and  to  account  for 
them  in  the  program  plan. 

B.  STUDY  OBJECTIVES.  The  study  objectives  are  to  identify  the  causes  of 
transition  problems  and  to  develop  a  production  risk  assessing  methodology 
which  addresses  and  quantifies  initial  production  uncertainties. 

C.  STUDY  APPROACH.  Research  began  with  a  review  of  pertinent  literature 
and  current  policy  regarding  production  planning  and  budgeting.  TRACE  and 
similar  techniques  were  reviewed  for  application  potential.  Selected  Major 
Subordinate  Commands  and  Project  Management  Offices  within  the  US  Army 
Materiel  Development  and  Readiness  Command  and  contractors  with  recent 
experience  in  moving  from  R&D  to  production  were  visited  to  gain 
insights  into  problem  areas.  A  methodology  was  then  developed  to  address 
production  uncertainties. 

D.  CONCLUSIONS  AND  RECOMMENDATIONS.  By  analyzing  Army  SAR  systems  during 
the  transition  from  R&D  to  production,  it  was  determined  that  a  35tf 
average  cost  growth  was  occurring  in  the  Procurement  appropriation 
(excluding  inflation  and  quantity  changes)  which  amounts  to  over  $5 
billion.  It  was  also  determined  that,  as  in  R&D,  risks  are  inherent  in  all 
systems  during  early  production  and  often  lead  to  large  production  cost 
increases.  The  various  techniques  used  to  address  risk  and  uncertainty, 
including  those  used  in  the  Army's  TRACE  program,  were  reviewed.  Although 
these  same  techniques  can  be  used  to  address  initial  production  risks, 

the  Production  Risk  Assessing  Methodology  (PRAM)  offers  an  improvement. 

It  combines  an  empirically  developed  risk  structure  with  conventional  cost 
estimating  techniques  to  quantify  initial  production  risks  into  a  dollar 
estimate  which  complements  the  Baseline  Cost  Estimate.  Using  the  PRAM, 
production  risk  ana  cost  growth  can  be  reduced.  However,  PRAM  should  first  be 
appropriately  tested.  Following  successful  testing,  it  should  be  incorporated 
on  Army  major  weapon  systems. 

E.  IMPLEMENTATION.  PRAM  has  been  tested  and,  with  some  modifications, 
incorporated  into  a  methodology  titled  Total  Risk  Assessment  Cost  Estimating 
for  Production  (TRACE/P).  As  an  approved  DARCOM  methodology,  TRACE/P  is 
being  applied  where  appropriate  to  develop  cost  estimates  which  include 
consideration  for  initial  production  risk. 
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CHAPTER  I 


INTRODUCTION 

A.  BACKGROUND/PROBLEM. 

The  Army  is  currently  moving  a  significant  number  of  sophisticated 
weapon  systems  from  Research  and  Development  (R&D)  to  production.  This 
transition  process  is  surfacing  a  considerable  number  of  production 
problems.  Some  of  the  problems  are  anticipated,  but  many  others  are  not. 
These  unexpected  problems  usually  result  in  undesirable  funding  adjustments 
and  what  is  perceived  as  program  cost  growth. 

The  reasons  for  the  difficulties  and  attendant  cost  growth  are  many 
and  varied.  Sometimes  the  problems  are  beyond  the  control  of  those 
responsible  for  managing  the  system;  nonetheless,  they  occur  regularly. 

A  methodology  is  needed  to  analyze  the  uncertainties  of  entering  production 
and  to  account  for  the  risk  impacts  in  the  program  plan.  By  planning  and 
budgeting  for  initial  production  risks,  program  disruptions  and  costs  can 
be  minimized. 

Two  recent  memorandums  within  DOD  have  recognized  this  problem  and 
initiated  efforts  for  improvement.  They  are  DEPSECDEF  Memorandum, 

30  Apr  81,  subject:  Improving  the  Acquisition  Process  (Recommendations 
6  and  11)  and  the  VCS,  Army  Memorandum,  22  Jul  81,  subject;  Cost 
Discipline. 

One  technique  currently  accepted  by  Congress  and  used  to  address 
uncertainties  in  R&D  is  the  Total  Risk  Assessing  Cost  Estimate  (TRACE). 

This  technique  assesses  technical,  cost,  and  schedule  risks  and  establishes 
a  TRACE  deferral  (funds  budgeted  for  uncertain  activities)  that  is 
available  for  the  Project  Manager  (PM)  to  draw  upon  when  justified. 


The  DARCOM  MSC's  and  PMO's  visited  to  gain  an  understanding  of 
typical  early  production  problems  are: 

1.  US  Army  Aviation  Research  &  Development  Command  (AYRADCOM) 

a.  CH-47  Mod 

b.  Remotely  Piloted  Vehicle  (RPV) 

c.  Advanced  Attack  Helicopter  (AAH) 

d.  Blackhawk 


2.  US  Army  Missile  Command  (MICOM) 

a.  Stinger 

b.  Pershing  II 

c.  Roland 

d.  Patriot 

3.  US  Army  Tank-Automotive  Command  (TACOM) 

a.  Abrams  Tank  (Ml) 

b.  Fighting  Vehicle  System  (FVS) 

The  defense  contractors  visited  are: 

1.  Chrysler 

Lima  Army  Tank  Plant 
Lima,  Ohio 

2.  Food  and  Machinery  Corporation  (FMC) 

San  Jose,  California 

3.  General  Dynamics 
Pomona,  California 

Additional  government  organizations  involved  in  transition  related 
activities  were  also  visited  to  gain  insights  into  initial  production 
problems  and  uncertainties. 

D.  REPORT  ORGANIZATION. 

Chapter  II  describes  the  transition  period  and  recent  transition 
problems  and  cost  growth  experience  within  the  Army.  The  techniques 
currently  available  to  assess  program  risk  are  discussed  in  Chanter  III, 
and  Chapter  IV  presents  a  specifically  designed  methodology  for  assessing 
initial  production  risk.  Chapter  V  states  the  study  conclusions  and 
implementation  recommendations. 


CHAPTER  II 


TRANSITION  FROM  DEVELOPMENT 
TO  PRODUCTION 


A.  INTRODUCTION. 

A  key  milestone  in  the  life  cycle  of  an  Army  weapon  system  is  the 
decision  to  enter  production.  Milestone  III,  the  production  decision 
point,  occurs  years  after  concept  exploration  begins,  but  only  a  small 
percentage  of  the  total  expected  program  life  cycle  cost  has  been  incurred 
up  to  this  point.  Most  of  the  program  funds  remain  to  be  expended  during 
the  Production/Deployment  Phase  following  Milestone  III.  Typically,  R&D 
accounts  for  approximately  15?  of  the  program  cost,  production  30?, 
and  operation  and  support  the  remaining  55?. [3]  Therefore,  it  is  prudent 
that  the  risks  of  entering  production  be  understood  as  clearly  as 
possible  before  the  program  is  committed  further. 

This  study  focuses  on  the  uncertainties  and  problems  experienced 
during  the  transition  period  from  development  to  production.  It  begins 
with  those  activities  and  events  occurring  perhaps  two  or  three  years 
before  Milestone  III  and  extends  to  the  state  of  routine  production. 

Although  there  is  no  commonly  accepted  definition  of  this  transition  period, 
it  is  depicted  in  Figure  2.1  as  the  shaded  area  with  Milestone  III  roughly 
in  the  middle. [20] 

The  Army  attempts  to  manage  and  to  prepare  for  eventual  production 
through  many  activities,  some  starting  early  in  the  life  cycle.  The  study 
bibliography  contains  references  which  describe  these  activities  and  the 
system  life  cycle  of  which  the  transition  period  is  a  part.  They  are  not 
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repeated  here  except  to  mp"tis.-.  two  of  the  most  effective  and  well  known: 
(a)  IVoduci  bi  1  i  ty  Engineering  and  Planning  (PEP),  and  (b)  Production 
Readiness  Reviews  (PRR).  Figure  2.1  shows  the  relative  timinq  of  these 
activities  during  the  system  life  cycle. 

Both  PEP  and  PRR's  are  conducted  during  Full  Scale  Engineering 
Development  (FSED)  to  minimize  production  risks  and  to  ensure  a  smooth 
transition  from  development  to  production.  These  activities  are  described 
as  foil ows : 

1 .  Produci bi 1 ity  Engineering  and  Planning  (PEP) . 

This  element  includes  cost  incurred  in  assuring  the 
produci bil ity  of  the  developmental  weapon  system,  item,  or 
component.  PEP  involves  the  engineering  tasks  necessary 
to  insure  timely,  efficient  and  economic  production  of 
essential  material  and  is  primarily  software  in  nature. 

PEP  includes  efforts  related  to  development  of  the  Techni¬ 
cal  Data  Package  (TOP),  Quality  Assurance  (QA)  plans,  and 
special  production  processes  to  assess  produci bi 1 ity.  Also 
included  are  development  of  unique  processes  essential  to 
the  design  and  manufacture  of  the  materiel  and  details  of 
performance  ratings,  dimensional  and  tolerance  data,  manu¬ 
facturing  assembly,  sequences,  schematics,  mechanical  and 
electrical  connections,  physical  characteristics  including 
form,  fit  and  finishes,  inspection  test  and  evaluation 
requirements,  calibration  information,  and  quality  con¬ 
trol  procedures . [9] 

2 .  Product. ion  Readiness  Review  (PRR) . 

A  formal,  documented,  systematic  review  of  a  proqran, 
to  find:  (a)  If  the  system  design  is  ready  for  pro¬ 
duction;  (b)  Production  engineering  problems  have  been 
identified  and  solutions  are  in  progress;  (c)  Quality 
assurance  and  acceptance  test  procedures  are  adequate; 
and  (4)  The  Army  and  producer  have  adequately  planned 
for  the  production  phase. [8] 

In  spite  of  the  Army's  efforts  to  prepare  for  production  and  to  control 
system  cost,  a  weapon  system  typically  experiences  problems  durinq  this 
transition  period  which  eventually  contribute  to  substantial  increases  in 
program  cost.  Figure  2.2  shows  the  extent  and  alarming  trend  of  the  total 
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IN  THE  PROCUREMENT 
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FIGURE  2.2  ARMY  WEAPON  SYSTEM  COST  GROWTH 
(SOURCE:  STUDY  OF  ARMY  LOGISTICS,  1981,  AND  ARMY  SAR’S) 


increase  in  RDT&E  and  Procurement  appropriations  for  Army  systems  reported 
in  the  Selected  Acquisition  Report  (SAR).[34]  Excluding  inflation,  there 
still  has  been  a  26?i  increase  in  the  procurement  appropriation  from  the 
first  quarter  of  FY  78  through  the  third  quarter  of  FY  81  compared  to  15% 
increase  in  RDT&E  during  the  same  period. 

Design  uncertainties  and  R&D  problems  are  well  recognized.  Production 
uncertainties  and  problems  were  not  as  well  recognized  until  recently, 
but  they  are  similar  and  have  a  much  larger  risk  impact  on  program  cost. 
Figure  2.2  shows  the  effect  both  as  a  percentage  increase  and  in  absolute 
dollars.  Also,  in  a  review  of  the  cost  growth  during  the  1970's  of  31 
DOD  systems,  RAND  found  that  most  of  the  systems  that  had  passed  DSARC  III 
(i.e.,  Milestone  III)  exhibited  growth  in  both  the  development  and 
production  phase. [13]  The  need  for  more  attention  to  production  and  for 
better  planning  and  control  of  initial  production  uncertainties  is  clear. 

B.  GENERAL  COST  GROWTH. 

Much  has  been  written  about  the  causes  of  cost  growth  throughout 
the  life  cycle  of  a  weapon  system.  The  Institute  for  Defense  Analyses 
(IDA)  summarized  some  of  the  more  frequently  cited  causes  of  cost  growth 
as  follows:[l] 

"Force  Majeure" 

•  Natural  disaster 

•  Civil  disorder 

•  Labor  strike 

•  Fire 

General  Economic  Inflation 

Cost  estimates  based  on  previous  similar  system  (each  succeeding 

generation  tends  to  cost  more  than  last  generation). 

Supply  shortages 

Labor  shortages 

Poor  management 
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Technological  uncertainty 

•  Unknowns 

•  Unknown  unknowns 
Environmental  laws/regulations 
Specification  changes 
Quantity  changes 

Rel iabil ity  problems 

Concurrency  (trying  to  produce  too  fast) 

Tight  budgets 
Competitive  environment 

•  within  branch  of  service 

•  within  service 

•  among  services 

•  DOD  vs.  other  federal  agencies 

•  Executive  branch  vs.  Congress 

•  among  contractors 

•  among  individuals 

IDA  emphasized  that  two  causes  should  be  singled  out  because  of  their 
impact: 


1.  The  competitive  environment  in  which  weapon 
systems  are  developed  is  the  major  factor  leading 
to  cost  growth;  and 

2.  Tight  budgets  are  an  often  overlooked  cause 
of  cost  growth. 

RAND,  in  an  earlier  study,  examined  DOD  major  weapon  systems  cost 

growth.  Using  the  SAR  cost  variance  categories,  RAND  found  that 

"The  persistance  of  cost  growth  after  DSARC  III  can 
be  traced  to  two  principal  causes:  schedule  slip¬ 
pages  and  efforts  to  increase  system  performance. "[13] 

Although  DOD  may  attempt  to  shift  production  risk  to  a  defense 
contractor  through  the  use  of  firm,  fixed  price  (FFP)  contracts,  the 
Army  Procurement  Research  Office  (APRO)  has  shown  in  earlier  work  that 
this  has  little  impact  on  cost  growth.  "FFP  contracts  suffered  a  net 
53%  cost  growth  -  almost  identical  to  the  entire  sample  cost  growth. "[22] 
Nearly  all  of  the  FFP  contracts  were  for  production. 
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In  summary,  the  cost  growth  literature  within  the  defense  community 
agrees  that  weapon  system  costs  during  R&D  and  production  are  increasing 
for  many  and  varied  reasons. 

c •  transition  cost  growth. 

The  magnitude  and  trend  of  transition  problems  were  studied  further 
in  terms  of  budgetary  impact.  In  1979,  Augustine  reported  that  for 
38  DOD  programs  from  1962-1976,  program  cost  growth  of  9%  occurs  after 
R&D  is  complete,  adjusting  for  inflation  and  quantity  changes. [2] 

Factoring  out  the  R&D  cost,  procurement  cost  growth  for  these  SAR 
programs  was  about  12%. 

To  evaluate  current  programs,  eleven  Army  SAR  systems  that  have 
recently  undergone  transition  from  R&D  to  production  were  analyzed  to 
determine  Procurement  cost  growth  during  early  production.  SAR  Procure¬ 
ment  cost  data  in  constant  dollars  was  adjusted  for  quantity  changes 
with  a  baseline  taken  at  the  quarterly  SAR  three  to  six  months  prior  to 
ASARC  III.  The  systems  studied  and  summary  results  are  shown  in  Table  2.1. 
The  magnitude  of  growth,  averaging  35.5%  and  totaling  over  $5  billion, 
indicates  an  unfavorable  trend  of  higher  growth  in  recent  years. 

To  normalize  the  growth  rate  to  reflect  differences  in  the  length 
of  time  these  systems  haye  been  in  production,  an  Average  Quarterly 
Growth  Rate  was  computed  based  on  the  equation 

1  +  Total  Percentage  Growth  =  (1  +  i)n 

where:  Total  Percentage  Growth  is  the  percent  increase  from  the  baseline 
cost , 

i  is  the  Average  Quarterly  Growth  Rate,  and 
n  is  the  number  of  quarters  of  SAR  data  from  the  baseline. 

The  Average  Quarterly  Growth  Rate  is  also  shown  on  Table  2.1. 


TABLE  2.1  SELECTED  ACQUISITION  REPORT  DATA  PROCUREMENT  COSTS 
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WEIGHTED  AVERAGE  35.5 


The  causes  of  cost  growth  reported  in  the  SAR's  are  shown  in  Table  2.2. 
It  indicates  "estimating  changes"  as  the  primary  contributor.  However, 
because  of  the  guidance  followed  in  classifying  SAR  changes,  "estimating 
changes"  include  cost  growth  beyond  that  which  is  caused  only  by 
optimistic  estimates. 

Despite  the  limitations  of  SAR  cost  change  classifications, [17,  13], 
recurring  patterns  can  be  identified  and  conclusions  can  be  drawn  by 
observing  the  timing  and  cost  growth  reported  in  the  SAR.  By  measuring 
SAR  Procurement  constant  dollar  growth,  adjusted  for  quantity  changes, 
from  a  baseline  (t  =  0  months)  from  the  second  SAR  prior  to  ASARC  III, 
the  pattern  exhibited  in  Figure  2.3  emerged.  Most  of  the  overall  35.5^ 
growth  occurs  in  the  first  24  months  of  the  time  period  considered,  with 
very  steep  rises  between  3-9  months  (ASARC  III  time  frame)  and  18  -  24 
months  (1  -  1  1/2  years  after  ASARC  III). 

Figure  2.4,  which  measures  quarterly  growth,  shows  cost  growth 
occurring  in  approximately  annual  cycles  with  the  greatest  magnitude  in 
the  first  two  years.  Cyclic  growth  can  continue  beyond  the  first  two 
years  though,  as  demonstrated  in  Figure  2.7.  The  procurement  cost 
growth  in  Army  systems  may  be  more  severe  than  indicated  thus  far  in 
Army  SAR's  since  most  of  the  eleven  systems  have  not  yet  reached 
production  maturity.  The  timing  of  reported  growth  seems  to  correlate 
with  the  updated  cost  estimates  prepared  for  ASARC  III  (t  =  6  months)  and 
the  annual  contract  negotiation  and  award  cycles. 

The  cost  growth  pattern  exhibited  by  the  aggregate  of  the  eleven 
systems  is  not  evident  when  analyzing  each  individual  system. 
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TABLE  2.2  SAR  COST  VARIANCE  REASONS 


CHANGE 

CAUSE 

AMOUNT 

$M 

PERCENTAGE 

ECONOMIC 

N/A 

N/A 

QUANTITY 

-482 

-10.4 

SCHEDULE 

398.5 

8.6 

ENGINEERING 

361 .9 

7.8 

SUPPORT 

701.8 

15.2 

ESTIMATING 

3635.8 

78.8 

OTHER 

0 

0 

TOTAL 

4616 

100 

(Source:  11  Army  SAR  Systems;  Time  period  of  3-6  months  prior  to  ASARC  III 
to  June  1981  in  Constant  $) 
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1981  SAR 


FIGURE  2.4  AVERAGE  QUARTERLY  PROCUREMENT  COST 
GROWTH  FOR  ARMY  SAR  SYSTEMS 


Figures  2.5,  2.6,  and  2.7  point  out  that  each  program  has  a  unique  set 
of  problems  that  are  reflected  in  cost  growth  patterns  of  varying  degree 
and  periodicity.  Some  programs  grow  from  the  beginning,  others  delay 
growth  until  later  in  the  production  cycle,  and  others  have  experienced 
little  growth.  This  is  consistent  with  the  notion  of  uncertainty  and 
probabilistic  occurrences.  Statistical  techniques,  although  applied  to 
individual  programs,  are  more  meaningful  on  an  aggregate  basis. 

D.  TRANSITION  PROBLEMS/UNCERTAINTIES. 

This  study  has  attempted  to  identify  the  fundamental  or  root  causes 
of  production  cost  growth  and  to  understand  the  relationships  among  the 
various  reasons  claimed  for  cost  growth  in  the  systems  reviewed. 

Production  problems  and  their  reported  reasons  may  not  reflect  the  root 
cause  of  cost  growth.  Rather  than  being  truly  isolated,  many  of  the 
reasons  offered  can  be  thought  of  as  part  of  a  chain  which  starts  some¬ 
where  (i.e.,  the  root  cause)  and  eventually  leads  to  cost  growth. 
Production  problems  that  are  not  root  causes  are  merely  precipitates  of 
the  root  cause;  although  the  problems  may  truly  reflect  root  causes  in 
other  paths  at  other  times. 

For  example,  PMO  and  contractor  personnel  cited  many  different  reasons 
for  transition  cost  growth,  and  not  all  were  root  causes.  The  following 
are  representative  of  the  frequently  cited  reasons  for  transition 
problems : 
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FIGURE  2.5  CUMULATIVE  PROCUREMENT  COST  GROWTH 
(SOURCE:  ARMY  SAR'S) 
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FIGURE  2.6  CUMULATIVE  PROCUREMENT  COST  GROWTH  (2) 
(SOURCE:  ARMY  SAR'S) 
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FIGURE  2.1  CUMULATIVE  PROCUREMENT  COST  GROWTH  (T) 


a.  'les  igr,  instability 

1'.  new  production  facility 

c.  inadequate  PEP 

d.  production  rate  changes 

e.  management,  problems 

f.  sole  source  contracting 

g.  optimistic  estimates 

h.  new  production  processes 

i.  software  problems 

j.  PIP'S 

k.  lack  of  skilled  people 

l.  workforce  instability 

m.  inflation 

n.  testing  problems 

o.  long  lead  items 

p.  production  stretchouts 

q.  requirements  changes 

r.  concurrency 

s.  inadequate  faci 1 i ti zation 

t.  quantity  changes 

u .  po 1 i t i c  s 

v.  shortage  of  specialized  contractors 

w.  budgetary  constraints 

x.  inaccurate  learning  curves 

y.  workarounds 

z .  exot i c  mat eri al s 

But  more  likely  than  not,  production  stretchouts  or  quantity  changes,  for 
example,  were  merely  precipitates  of  something  else  -  inflation,  perhaps. 
It  is  believed  that  understanding  the  relationships  among  the  root  causes 
of  transition  problems  and  cost  growth  is  key  to  developing  a  useful 
methodology  for  assessing  initial  production  risk. 

Based  upon  this  analysis  and  a  recognition  that  past  problems  identify 
potential  future  risks,  a  risk  structure  was  empirically  developed  to 
address  initial  production  risk.  There  are  numerous  ways  to  categorize 
and  relate  production  risk.  This  study's  criteria  were  that  the  structure 
not  only  capture  and  isolate  the  fundamental  risk  areas,  but  also  main¬ 
tain  as  much  independence  as  practical  among  the  individual  areas.  This 
facilitates  the  statistical  treatment  of  cost  in  the  methodology  developed 
in  Chapter  IV. 
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Figure  2.8  shows  the  empirically  developed  structure  for  initial 
production  risk.  There  are  three  major  groups:  (a)  External,  (b) 

Resource,  and  (c)  System.  Individual  risk  categories  exist  within  each 
group.  The  Resource  and  System  groups  are  considered  to  be  internal  to  the 
PMO.  Resource  risks  relate  to  the  production  activity  itself  and  resources 
needed  to  produce  the  system.  System  risks  are  product  oriented  and  stem 
primarily  from  inherent  technological  and  design  risks  carried  over  into 
production.  External  risks  are  those  outside  the  control  of  a  PMO 
primarily  in  the  requirements  area.  Chapter  IV  presents  a  detailed  dis¬ 
cussion  of  each  group  and  the  individual  risk  categories. 

Using  the  risk  structure  in  Figure  2.8,  the  problems  experienced  by 
selected  Army  systems  during  the  transition  period  are  summarized  as  shown 
in  Table  2.3.  An  "X"  in  a  cell  indicates  that  a  problem  was  cited  by  a  key 
individual (s)  in  the  PMO  or  prime  contractor.  The  problems  cited  are^ 
accommodated  by  the  respective  risk  categories.  Table  2.3  only  indicates 
the  relative  frequency  of  problem  occurrence  based  on  the  systems  reviewed. 
But,  the  fact  that  the  risk  structure  accommodates  the  e*sons  i'-.'  .  ost 
growth  as  seen  by  key  experienced  individuals  is  important  for  the  success¬ 
ful  development  of  a  production  risk  assessing  methodology. 

In  an  attempt  to  gain  a  more  indepth  understanding  of  the  responses, 
problems  cited  were  arrayed  by  risk  category  on  an  individual  basis, 

Table  2.4.  The  number  in  each  cell  represents  the  number  of  key  individuals 
within  a  PMO  or  prime  contractor  who  cited  a  problem  relating  to  the  risk 
category  as  a  reason  for  Procurement  cost  growth.  However,  it  must  be 
pointed  out  that  since  the  opportunities  for  discussion  with  personnel 
within  each  system  were  not  uniform,  the  strength  of  association  between 
frequency  of  response  and  degree  of  impact  could  not  be  determii  '.  The 
time  allotted  and  method  employed  did  not  permit  the  quantification  of 
impact  intensity. 
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FIGURE  2.8  INITIAL  PRODUCTION  RISK  CATEGORIES 


TABLE  2.3  SUMMARY  OF  TRANSITION  PROBLEMS 
EXPERIENCED  BY  SELECTED  ARMY  SYSTEMS 


C.  Performance 


TABLE  2.4  TABULATION  OF  TRANSITION  PROBLEM 

CITATIONS 

SYSTEM 

FOR  SELECTED  ARMY  SYSTEMS 

CITATION 

ATEGORY  ^ 

A  B  C  D  E  F  G  H  I  J  FREQUENCY 

EXTERNAL 

A.  Management  Goals 
8.  Inflation 
C.  Unknowns 


RESOURCE 

A.  Management 

B.  Funds 

C.  Material 

D.  Facilities 

E.  Labor 


0  7  7 
3  5  0 
3  5  7 


SYSTEM 

A.  Design  Stability 

B.  Producibility 

C.  Performance 


CHAPTER  III 


RISK  ASSESSMENT 

A.  INTRODUCTION. 

It  is  commonly  recognized  that  an  R&D  effort  implies  risk.  The  Army 
has  developed  a  procedure  using  TRACE  to  incorporate  risk  analysis  into 
the  RDT&E  budgeting  process. 

An  analysis  of  pertinent  literature  and  personal  interviews  indicate 
there  is  also  significant  risk  associated  with  the  transition  from  R&D 
into  production.  Therefore,  TRACE  and  similar  techniques  were  investigated 
for  possible  application  to  initial  production  risks. 

B.  TRACE  FOR  R&D. 

1 .  Concept . 

The  TRACE  concept  was  formulated  to  provide  realistic  cost 
estimates  and  thereby  minimize  subsequent  disruptive  reprogramminq  actions. 

It  was  desinned  to  minimize  cost  overruns  without  resorting  to  qross 
overbudgeting.  Specific  emphasis  is  on  allocation  of  funds  to  reduce  the 
cost  growth  effects  resulting  from  the  occurrence  of  events  that  could  not 
be  programmed  because  of  the  lack  of  certainty  that  they  would  material ize. [37] 

It  is  impossible  to  exactly  predict  the  cost  of  a  project.  In  reality, 
the  final  cost  of  a  project  will  fall  somewhere  within  a  distribution  as 
shown  in  Figure  3.1.  TRACE  is  a  point  estimate  selected  from  that  dis¬ 
tribution  such  that  its  probability  of  being  exceeded  is  at  an  acceptable 
level.  Figure  3.1  shows  the  TRACE  as  having  an  equal  (50/50)  probability 
of  underrun  and  overrun, but  another  point  such  as  60/40  or  70/30  could 
have  been  selected.  When  the  Baseline  Cost  Estimate  (BCE)  is  placed  on 
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this  distribution  of  possible  costs,  it  will  fall  to  the  left  of  the 
TRACE.  The  BCE  is  calculated  from  engineering  estimates  of  specifically 
programmed  activities,  and  although  it  generally  includes  some  contingency, 
it  does  not  include  consideration  for  many  uncertain  activities  that  are 
statistically  probable.  Inis  is  further  illustrated  in  Figure  3.2. [381 

2.  Mechanics. 

The  PMO  calculates  a  BCE  and  a  TRACE  and  submits  them  through 
channels  to  Headquarters,  Department  of  the  Army  (HQDA).  The  BCE 
represents  project  target  cost.  The  TRACE  is  used  for  programming/ 
budgeting  and  as  the  cost  entry  in  the  Five  Year  Defense  Plan  (FYDP ) . 

Upon  budget  approval  by  DOD/Congress ,  the  BCE  amount  is  released  to  the 
PM  for  program  execution.  The  remainder  known  as  "TRACE  deferral,"  is 
retained  by  HQDA  to  serve  as  a  source  of  funds  for  the  PM  to  draw  upon 
when  justified  to  accomplish  additional  uncertain  activities. 

The  retention  of  TRACE  deferral  funds  by  HQDA  allows  for  managerial 
control  and  possible  cost  savings.  These  deferral  funds  are  on  a  line 
item  and  fiscal  year  basis,  and  each  year's  funds  are  available  for 
obligation  for  two  years.  During  the  fifth  quarter  of  availability,  the 
PMO  must  decide  whether  to  release  the  funds  to  HQDA  for  reproqramming. 

The  funds  will  be  automatically  reprogrammed  if  no  action  is  taken  by 
the  seventh  quarter.  Any  request  by  the  PM  to  obtain  deferral  funds  must 
be  accompanied  by  sound  justification  within  established  guidelines. 

For  example,  funds  will  not  be  used  to  offset  costs  of  major  requirements 
changes  which  instead  will  be  accommodated  by  restructuring  the  proqram 
and  recomputing  the  TRACE. 
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3.  Lessons^  Learned . 

TRACE  for  RDT&E  has  been  applied  by  the  Army  since  the 
6  March  1975  "Letter  of  Instruction  (LOI)  for  Implementation  of  RDTK  cost 
Realism  for  Current  and  Future  Development  Programs."  Between  FY  76  and 
FY  84,  TRACE  has  been  applied  to  31  programs.  As  of  FY  81,  TRACE  has 
been  credited  with  avoiding  17  reprogramming  actions  and  avoiding  27 
Congressional  approvals. [12]  The  number  of  programs  using  TRACE  and  the 

amount  of  TRACE  funding  have  increased  as  shown  in  Figure  3.3.  Those 

* 

programs  that  have  more  recently  used  TRACE  have  relied  on  TRACE  as  a 
greater  percentage  of  RDTE  funding  in  the  TRACE  funded  years  than  earlier 
TRACE  programs,  as  shown  in  Figure  3.4.  Thus  TRACE  for  R&D  has  become 
generally  accepted  and  useful. 

TRACE  for  R&D  does  have  some  problems  however,  that  should  be 
addressed  as  potential  pitfalls  for  application  to  initial  production. 

a.  Initially,  PM's  were  hostile  to  the  concept  of  project  funds 
being  held  in  reserve  by  DA. [40] 

b.  In  theory,  approximately  one-half  of  the  programs  usino  TRACE 
should  not  need  all  of  their  programmed  TRACE  funds  and  the  other  half 
should  need  more  than  the  amount  programmed.  In  reality  though,  very 
few  (5  times  out  of  88  potential)  programs  from  FY  76  -  FY  81  have 
turned  in  unused  TRACE  funds  in  a  fiscal  year.  Two  possible  explanations 
are  that  TRACE  may  have  been  historically  underestimated  (or  underfunded) 
or  that  TRACE  becomes  a  "self-fulfilling  prophecy." 

c.  TRACE  methodology  has  been  criticized  as  too  subjective,  too 
resource  demanding,  too  restrictive,  and  too  inflexi ble. [181 
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d.  TRACE  has  not  eliminated  RDTE  overruns . [32 ,  34] 

4.  Methodology. 

A  survey  of  20  Army  PMO's  that  had  TRACE  experience  found  that 
three  principal  methodologies  were  used  to  compute  TRACE,  with  some 
unique  variations. [18]  The  three  methods  were  classified  as  Risk 
Percentage,  Risk  Factor,  and  Probabil istic  Network  Models.  A  fourth 
method,  called  Probabi 1 i Stic  Event  Analysis  or  Risk  Tabulation  was  not 
found  to  be  used  by  any  PM.  These  four  methodologies  are  as  follows:[3Ql 


a.  Risk  Percentage. 


The  Risk  Percentage  method  is  an  undocumented  procedure  in 
which  the  TRACE  is  computed  by  adding  a  percentage  (10  -  15')  to  the 
BCE  at  a  summary  level.  The  percentage  is  subiectively  determined  by 
experts  based  on  past  experience,  risk  assessment  and  judgment.  Time 
phasing  is  accomplished  subjectively  or  assuming  proportional i ty  to  the 
RDTE  effort.  This  method,  although  used,  has  not  been  formally  approved 


because  of  similarities  to  cont. .  ..cy  fund  or  management  reserve  concepts, 
b.  Risk  Factor. 


The  Risk  Factor  approach  described  in  the  1975  LOI  computes 
TRACE  by  assigning  risk  factors  to  discrete  Work  Breakdown  Structure  fWBS) 
elements  of  the  BCE.  After  estimating  WBS  element  costs  as  part  of  the 
RDTE  BCE,  each  element  is  assessed  for  uncertainty  by  experts.  Historical 
data  on  previous  similar  systems  is  used  when  available,  but  most  risk 
factors  are  subjectively  determined.  A  factor  represents  the  cost  increase 
expected  for  a  WBS  element  as  a  result  of  technical  uncertainty  associated 
with  that  particular  element  plus  the  interrelated  uncertainties  associated 
with  other  WBS  elements  that  interface  with  it.  By  multiplying  each 
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WliS  cost  by  its  risk  factor,  a  TRACt  is  computed.  Time  phasing  is 
subjective  or  assumed  proportional  to  BCE  time  phasing. 

Advantages  of  this  approach  are:  (1)  analysis  does  not  require  a 
high  analytical  skill  level;  (2)  analysis  can  be  performed  quickly  and 
inexpensively  in  comparison  to  computer  modeling;  (3)  analysis  can  be 
easily  understood;  and  (4)  quality  of  analvsis  can  be  easily  evaluated 
by  management. 

The  most  serious  disadvantage  of  this  approach  is  in  the  determination 
of  the  risk  factors.  Because  of  the  apparent  simplicity  of  the  apnroach, 
there  might  be  a  tendency  to  use  the  risk  factor  as  simply  a  “fudge 
factor."  To  handle  the  factor  in  such  a  manner  would  reduce  the  credibility 
of  the  cost  estimate.  In  addition,  the  factor  is  implicitly  assumed  to 
be  constant  for  each  element  throughout  duration  of  the  project. 
c •  Probabil istic  Network  Modeling . 

Probabil istic  Network  Modeling  is  a  combined  approach  using 
Program  Evaluation  and  Review  Technique  (PERT)  principles  and  Monte  Carlo 
simulation  techniques.  Various  computerized  models  can  be  used  for  this 
application,  including  VERT,  RISCA,  and  R I SNET . L 39 ]  An  RftD  oroqram  is 
first  displayed  as  a  network  of  interrelated  events  and  activities.  Cost, 
schedule  and  technical  uncertainties  associated  with  the  various  activities 
are  then  estimated.  The  model  iteratively  simulates  the  activities  and 
events  to  produce  time  -  phased  cost  and  schedule  distributions  for  the 
program.  The  analyst  can  adjust  the  TRACE  to  levels  reflecting  desired 
probability  of  cost  overrun  vs.  cost  underrun.  This  method  is  the  most 
rigorous  and  resource  demanding,  but  it  is  the  most  precise  and  risk 
inclusive. 
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Advantages  of  this  approach  are:  (I)  explicit  consideration  is  given 
to  activity  interaction;  (2)  the  TRACE  may  be  selected  from  the  total 
cost  distribution  as  output;  (3)  the  form  and  collection  of  output  is 
flexible;  (4)  the  model  can  be  easily  modified  and  rerun  to  answer 
"what- if?"  questions;  (5)  the  network  can  estimate  the  BCE  by  fixing 
schedules  and  removing  uncertai n/cont i ngency/fal 1  back  activities; 

(6)  the  network  can  serve  as  a  management  tool  to  track  and  control  ,  as 
well  as  predict  time  and  schedule;  and  (7l  the  network  can  be  used  to 
satisfy  the  Decision  Risk  Analysis  requirement. 

Disadvantages  of  this  approach  include:  (1)  a  high  skill  level  is 
required  to  build  the  network  and  collect  data;  {?.)  the  output  can  be 
sensitive  to  the  network  logic;  (3)  it  is  difficult  to  reconcile  this 
approach  with  the  WBS;  (4)  the  cost  is  initially  high;  and  (5)  it 
requires  considerable  data  collection. 

d .  Probabi 1 istic  Event  An a  lysis . 

Probabilistic  Event  Analysis,  or  Risk  Tabulation,  was  developed 
by  John  M.  Cockerham  and  Associates,  Inc.  to  correct  perceived  deficiencies 
in  the  Risk  Factor  approach.  Risk  for  each  WBS  element  is  separated  into 
two  categories  in  an  effort  to  assess  interactive  effects  between  WBS 
elements.  Internal  (stand  alone  or  isolated)  risks  are  assessed  as  well 
as  the  external  (interactive)  risks.  Using  conditional  probability 
theory,  the  overall  program  cost  risk  is  tabulated.  Uncertainties  assessed 
as  probability  values  are  determined  essentially  in  the  same  manner  as 
risk  factors.  Time  phasing  can  be  incorporated  into  the  calculation  by 
estimating  when  the  various  risks  will  occur  for  each  WBS. 


Ad v ci ntages  of  this  approach  are  that  it  is  relatively  easy  to  use, 
and  it  addresses  interaction  between  elements  so  it  should  give  better 
results  than  the  Risk  Factor  approach,  fii sadvantaqes  are  (1)  it  is 
highly  dependent  upon  the  skill  of  the  analyst  to  identify  and  account 
for  the  various  interdependencies;  (2)  it  is  sensitive  to  errors  in 
subestimates  and;  (3)  the  TRACF  uses  t he  BCE  as  a  basis  and  is  subject 
to  the  same  bias  as  the  BCF . 

C .  IN VTSTME.NJ.  PHASE.  MANAGEMENT.  RESERVE..  •'  I PMRh 

In  addition  to  the  TRACE  techniques  applied  to  the  R^D  phase,  a 
separate  approach  has  been  developed  and  applied  on  two  Army  system"  to 
establish  a  management  reserve  for  the  Investment  Phase. [BC] 

Uncertainty  elemencs  were  defined  separate  from  the  BCE  structure,  Ou* 
are  related  to  the  BCE  through  an  adjustment  to  the  Desiqn-to-Hni t- 
Production-Cost  (DTIJPC).  These  uncertainty  elements  are:  Production, 
Performance,  Siting,  Technology,  Resources,  Manaqement/Cont.rol  ,  Higher 
Management,  and  Other.  Triangular  cost  distributions  are  estimated  lot- 
each  uncertainty  element  through  subjective  assessments  of  maior  sub¬ 
system  r  s k  which  are  then  related  to  the  DT'JPC.  The  co<-f  distributions 
0rf-  combined  using  Venture  Evaluation  and  Review  Technique  (VFRT)  which 
generates  a  single  cost  distribution  with  a  mean  usually  exceeding  the 
DTUPC.  Fite  uiflerence  between  the  DTUPC  and  the  mean  (or  some  other  value 
from  the  single  cost  distribution)  is  the  basis  for  the  management  reserve. 

Advantages  ot  the  IPMR  approach  arc  (1)  the  uncertainty  elements 
worn  specifically  designed  ror  production;  (2)  the  analytical  techniques 
used  are  relatively  simple  to  explain;  (3)  it  is  flexible  in  that 


different  confidence  levels  for  risk  can  be  used;  and  (4)  the  triangular 
distribution  parameters  are  easily  estimated.  Disadvantages  are  (1) 
there  is  interdependency  among  the  uncertainty  elements;  (2)  it  requires 
accurate  DTUPC  estimates;  and  (3)  it  was  intuitively  developed  for  a 
specific  system. 


CHAPTER  IV 


PRODUCT  I OT!  RISK  ASSESSING  METHODOLOGY 

A.  INTRODUCTION. 

From  the  discussion  of  risk  and  TRACE  for  R&D  in  Chapter  III,  it 
appears  there  are  a  number  of  possible  ways  to  assess  initial  production 
risk.  The  approaches  for  R&D  TRACE  could  similarly  be  applied  to  the 
production  WBS  and  BCE.  The  Risk  Factor  approach,  for  example,  which  is 
popular  for  R&D  TRACE,  could  simply  be  used  to  adjust  the  BCE  cost 
elements  to  include  a  TRACE  amount  for  production.  But  the  BCE's  already 
include  some  considerat ion  of  risk  using  a  judgment  approach  similar  to 
the  Risk  Factor,  and  yet,  underfunding  continues  to  plague  programs. 

Recent  history  of  inaccurate  estimates  using  the  BCE,  as  evidenced  by 
the  Army  SAR's,  shows  something  more  is  needed  -  something  that  specifically 
addresses  production  risk  and  the  synergism  of  the  various  production 
cost  elements. 

One  approach  that  attempts  to  do  this  is  the  Investment  Phase  Manage¬ 
ment  Reserve  methodology .[36]  This  methodology  proceeds  in  the  right 
direction  but  can  also  be  improved  upon. 

B •  TAX0N0MY_0F_  INITIAL  PRODUCTION  RISKS . 

A  different  perspective  of  initial  production  risk  was  taken'  to 
improve  upon  the  shortcomings  of  the  current  techniques.  A  recognition 
that  past  production  problems  represent  risk  areas  for  future  systems 
suggested  an  empirically  developed  risk  structure  which  specifically 
addresses  initial  production  uncertainty.  To  develop  this  structure,  an 
extensive  list  of  initial  production  problems  recently  experienced  by 
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selected  Army  PMO's  was  compiled  from  field  interviews.  This  list  was 
supplemented  by  a  data  search  and  discussions  with  experienced  government 
and  contractor  personnel  outside  the  PMO's.  By  deductively  applying 
classical  production  management  theory  to  these  "real  world"  problems, 
the  major  risk  categories  evolved  as  illustrated  in  Figure  2.8. 

This  figure  illustrates  that  a  weapon  system,  constrained  by  limited 
resources ,  must  be  produced  within  a  dynamic  environment  of  external 
forces.  The  major  groups  of  "System  Risk"  and  "Resource  Risk"  are 
within  the  purview  of  the  project  manager/contractor,  while  the  group 
labeled  "External  Risk"  is  beyond  their  control.  The  arrows  in  Figure 
2.8  depict  the  relationship  between  these  three  categories  as  a  continuous 
two-way  flow  of  information. 

Figure  4.1  further  illustrates  these  relationships  as  a  taxonomy  in 
which  the  inner  circle  represents  system  risks,  the  middle  ring  represents 
the  resource  constraints,  and  the  outer  ring  represents  the  external 
risks.  Within  this  context,  the  three  major  groups  (System,  Resource, 
External)  were  further  subdivided  into  a  comprehensible  risk  categories 
as  described  below. 

1 .  System  Risk. 

The  ideal  situation  is  when  the  design  has  been  virtually 
stabilized  prior  to  initial  production,  all  aspects  of  producibil ity 
have  been  incorporated  into  the  technical  data  package  and  production 
line  items  subsequently  meet  all  performance  specifications.  In  reality, 
this  never  occurs  due  to  inherent  uncertainties  in  design  stability, 
producibil ity,  and  performance,  as  described  below. 


FIGURE  4.1  TAXONOMY  OF  INITIAL  PRODUCTION  RISK 


a .  Design  stabil ity. 

This  refers  to  the  level  of  design  stability  when  entering 
initial  production.  If  design  problems  have  not  been  resolved  during 
R&D,  the  attendant  uncertainties  will  carry  over  and  disrupt  initial 
production.  Indicators  of  risk  level  in  this  category  include  but  are 
not  limited  to: 

(1)  program  concurrency 

(2)  configuration  changes 

(3)  testing  requirements  and  results. 

b.  Producibi 1 i ty. 

It  is  essential  that  sufficient  attention  be  directed  to 
produci bi 1 ity  aspects  during  R&D;  otherwise,  the  result  will  be  severe 
disruptions  and  increased  cost  during  initial  production.  Indicators  of 
risk  level  in  this  category  include: 

(1)  adequacy  of  PEP  program 

(2)  adequacy  of  MM&T  program 

(3)  sophistication  of  manufacturing  processes 

(4)  custom  made  prototypes. 

c .  Performance . 

Problems  can  occur  with  system  performance  even  if  design  has 
stabilized  and  the  item  is  producible.  This  risk  area  includes  all  the 
"ilities"  (reliability,  availability,  maintainability,  dependability, 
capability,  etc.)  required  to  meet  rigid  performance  specifications.  If 
increasingly  sophisticated  performance  requirements  continue  to  drive 
weapon  designs,  performance  will  remain  an  area  of  high  uncertainty. 
Indicators  of  risk  level  in  this  category  include: 
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(1)  state-ot -the-art  technology 

(2)  exotic,  testing  requirements 

(3)  extent  ot  PIP  activity. 

2 .  Resource  Risk . 

The  system  requirements  of  design,  produc i bi 1 i ty ,  and  performance 
are  merely  academic  unless  resources  are  available  to  accomplish  them. 
These  resources  can  be  succinctly  divided  into  the  five  fundamental 
categories  of  material,  facilities,  labor,  funds,  and  management.  The 
increasing  demands  on  these  finite  resources  ensure  this  as  a  continuing 
risk  group. 

a.  Material. 

This  resource  refers  to  all  materials  and  purchased  parts 
going  into  the  weapon  system  and  its  direct  support  equipment.  It 
includes  raw  material  in  addition  to  specialized  vendor  items  such  as 
electronic  components,  engines,  transmissions,  etc.  Problems  may  occur 
at  the  prime  contractor,  subcontractor  or  vendor  levels.  Indicators  of 
risk  level  in  this  category  include: 

(1)  exotic/strategic  materials 

(2)  long  lead  times 

(3)  turbulence  in  specialized  vendor  industries. 

(4)  sole  source/proprietary  vendors 

b.  facJJ  itjes . 

This  resource  includes  brick  and  mortar  needs  as  well  as 
manufacturing  and  testing  equipment  and  tooling.  Facil itization  risk 
varies  depending  on  whether  existing  facilities  are  modified  or  new 
facilities  are  designed  and  built.  The  increasing  sophistication  of 
weapon  systems  brings  new  and  exotic  equipment  and  tooling  requirements. 


Indicators  of  risk  level  include: 

(1)  state-of-the-art  manufacturing  equipment 

(2)  manufacturing  equipment  lead  time 

(3)  new  versus  existing  manufacturing  facilities. 

c.  Labor. 

Labor  remains  a  major  contributer  to  weapon  s_>  cost. 
Competition  for  scarce  skills  in  such  critical  fields  as  engineering, 
software  design,  welding,  machining,  heavy  equipment  use  and  maintenance 
make  this  a  high  risk  category.  Indicators  of  risk  level  include: 

( 1 )  labor  availabil ity 

(2)  demand  for  scarce  skills 

(3)  training  requirements 

(4)  personnel  turnover. 

d.  Funds. 

Keen  competition  for  limited  funds  at  all  levels  of  government 
and  industry  from  departmental /corporate  level  to  project  level  causes 
uncertainty.  The  complicated  process  by  which  funds  are  estimated, 

requested,  appropriated  and  obligated  adds  to  the  uncertainty.  Inadequate 
funding,  if  even  for  a  short  period  of  time,  can  result  in  severe  program 
perturbations  that  ultimately  result  in  cost  growth.  Indicators  of  risk 
level  in  the  funding  category  include: 

(1)  optimism  in  cost  estimating 

(2)  timeliness  of  funds 

(3)  contractor  cash  flow. 

e .  Management. 

This  risk  category  includes  the  sufficiency  and  experience 
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of  management  personnel  in  both  the  contractor  and  project  management 
offices  and  general  manageability  of  the  project.  In  this  context 
management  is  just  as  much  a  resource  as  material,  facilities,  production 
labor,  and  funds.  Management  decisions,  particularly  in  the  planning 
and  controlling  functions,  have  tremendous  potential  cost  impact. 

Indicators  of  risk  level  include: 

(1)  management  turbulence 

(2)  relationship  between  PMO  and  contractor (s ) 

(3)  production  management  experience 

(4)  available  management  information  systems 

(5)  management  complexity. 

3 .  External  Risk . 

This  group  represents  uncertainties  over  which  program  management 
and  contractor  personnel  have  no  control.  These  uncertainties  constitute 
the  dynamic  environment  in  which  finite  resources  are  allocated  to  the 
production  of  many  systems.  The  three  major  categories,  as  described 
below,  are  management  goals,  inflation,  and  unknowns, 
a .  Management  _g o al_s . 

There  are  numerous  layers  of  management  above  the  program 
manager  that  are  continually  interpreting  threats  and  establishing  goals. 
These  include  cost,  schedule,  guantity  and  performance  goals.  The  resulting 
guidance  to  the  PMO  often  changes,  redirecting  the  program  and  causing 
perturbations  that  ultimately  cause  cost  growth.  Indicators  of  risk  level 
include: 

(1)  political  environment 

(2)  project  priority 

(3)  program/requirement  changes. 
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Inflation 

There  is  uncertainty  in  estimating  future  inflation  rates. 
Inaccurate  projections  of  inflation  can  result  in  substantial  program 
underfunding  which  contributes  to  apparent  cost  growth  in  two  ways. 

First,  the  underfunding  compared  to  actuals  is  generally  interpreted 
as  a  cost  overrun.  Second,  the  lack  of  funds  causes  program  disruptions 
which  result  in  redirection  of  resources  and  cost  growth, 
c .  U n  known  s . 

This  risk  category  includes  such  occurrences  as  natural 
disasters,  civil  disorders,  labor  strikes,  fires,  and  major  program 
setbacks  such  as  prototype  crashes  or  sabotage.  These  risks  cannot  be 
accurately  anticipated  but  history  shows  that  they  do  occur. 
c  •  PRODUCTION  RISK  ASSESSING  METHODOLOGY  (PRAM) . 

The  risk  structure  described  above  was  empirically  developed  from 
extensive  personal  interviews  and  a  literature  survey  of  the  causes  of 
"transition"  problems.  It  was  derived  independently  of  standard  cost 
categories  (i.e.,  the  BCE,  SAR,  etc.)  and  is  not  influenced  bv  their 
innate  biases.  The  structure  is  understandable  in  that  it  conforms  to 
traditional  production  categories.  It  is  flexible  because  risk  costs 
can  be  calculated  at  any  level  (i.e.,  system  vs.  subsystem);  it  is  not 
system  unique;  and  all  eleven  categories  need  not  be  used.  The  eleven 
basic  categories  are  reasonably  independent  if  care  is  taken  not  to  credit 
a  particular  risk  to  more  than  one  category.  The  structure  is  inclusive 
in  the  sense  that  all  potential  "transition"  problems  gleened  from  the 
interviews/literature  search  fit  into  the  eleven  categories. 
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Accepting  the  above  risk  structure,  an  attempt  must  be  made  to 
quantify  each  risk  category  as  it  applies  to  a  specific  system  and/or 
subsystem  and  relate  it  to  production  cost.  Obviously,  answering  the 
question  "how  much?"  for  each  risk  area  is  not  always  easy.  Some 
categories  such  as  Inflation  or  Facilities  are  relatively  easy  to 
quantify  using  the  BCE  data  as  a  base.  Others,  such  as  Produci hi  1 i ty 
or  Management,  are  much  more  difficult  and  must  rely  primarily  upon  sound 
judgment.  The  appropriate  cost  estimating  approach  for  a  risk  category 
should  be  used,  whether  it  is  the  Industrial  Engineering  approach, 
parametric,  analogy,  educated  guess  or  some  combination.  There  is  no 
"best  way"  to  quantify  all  risk  categories. 

This  union  of  conventional  cost  estimating  techniques  with  the  initial 
production  risk  structure  constitutes  a  methodology  to  assess  and  quantify 
production  risk.  Titled  Production  Risk  Assessing  Methodoloqy  (PRAM!, 
it  should  not  be  confused  with  a  direct  application  of  a  TRACE  technique 
to  production  risk. 

The  objective  of  the  PRAM  is  to  develop  individual  cost,  distributions 
for  those  high  risk  categories  pertinent  to  the  weapon  system  beino 
analyzed.  The  inclusion  of  any  risk  catenory  in  a  PRAM  exercise  is 
optional  since  unusual  production  risk  may  not  be  present  in  all 
categories  for  a  given  system  or  subsystem. 

If  subsystem  risks  are  to  be  analyzed  separately  and  then  combined 
into  a  system  level  individual  cost  distribution,  care  should  be  taken 
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tn  separate  the  risk  impact  as  appropriate  among  the  subsystems.  For 
example,  if  labor  is  a  high  risk  area  for  both  the  frame  and  propulsion 
subsystems,  care  should  be  taken  not  to  include  frame  labor  risk  in  the 
propulsion  labor  risk  assessment  and  vice  versa.  "Double  counting"  in 
the  risk  assessments,  whether  system  or  subsystem  level,  will  serve  to 
inflate  the  final  risk  cost  figures. 

Frequently,  the  individual  cost  distributions  can  be  described  with 
three  values  (low,  medium,  and  high)  and  assumed  to  follow  a  triangular 
probability  distribution.  However,  any  probability  distribution  can  be 
used,  as  long  as  it  is  appropr iately  descriptive  of  the  actual  risk. 

Figure  4.2  shows  the  summary  format  that  should  be  completed  from  risk 
cost  calculations  for  a  PRAM  exercise. 

The  cost  elements  in  the  BCF  should  be  used  where  appropriate  to 
assist  in  developing  the  risk-cost  distributions,  although  it  is  not 
necessary  in  all  cases. [7]  For  example,  Non-Recurring  Investment  (2.01) 
and  Sustaining  Tooling  (2.023)  relate  to  the  facilities  and  equipment  risk 
category;  Recurring  Engineering  (2.022),  Quality  Control  (2.024)  and  a 
portion  of  Manufacturing  (2.021)  relate  to  the  labor  risk  category; 
another  portion  of  Manufacturing  (2.021)  relates  to  the  materials  risk 
category;  and  Engineering  Changes  (2.03)  and  System  Test  and  Evaluation 
(2.04)  relate  to  the  design  stability  risk  category.  The  BCE  values 
should  be  noted  when  present  in  a  cost  distribution. 

Once  the  cost  distributions  are  determined  and  parameters  calculated 
for  the  appropriate  categories,  the  individual  risk  costs  can  be  calculated 
for  each  category  by  subtracting  the  BCE  value,  where  present,  from  the 
mean  PRAM  estimate.  Summing  the  individual  risk  costs  vields  the  system 
Total  Expected  Risk  Cost. 
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TOTAL  EXPECTED  RISK  COST  SUMMARY 
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The  Total  Expected  Risk  Cost  i *  he  mean  or  expected  v.u'ue  of  ihe 
convolved  risk  i.o  ,t  distribution.  If  ■  ■  ri.ir  t  i  u/ 1  ar  level  of  'oniidence 
is  desired,  the  variance  of  the  risk  c o s i  distribution  must  also  In- 
determined  and  used  to  adjust  the  total  risk  cost.  Hue  to  independence , 
the  variance  ,an  be  calculated  by  summing  the  variances  ot  the  individual 
cost  distributions,  it  can  also  be  determined  by  combining  the  estimated 
PRAM  distributions  using  techniques  such  as  VERT. 

Once  the  total  risk  cost  is  determined,  tne  Production  Risk  Assessing 
Cost  Estimate  (PRACE)  can  be  calculated.  PRACE  is  defined  os  the  sum  of 
the  BCE  Investment  Cost  and  the  total  risk  cost  just  as  TRACE  is  the 
sum  of  the  BCE  for  ROTXE  and  the  RXD  risk  cost.  It  should  be  noted  that 
the  BCE  values  shown  on  the  Total  Expected  Risk  Cost  summary  account  for  onlv 
the  appropriate  portion  of  the  total  BCE  Investment  Cost.  Also,  based  on  the 
cost  growth  experience  snown  in  Figure  2.4,  the  total  risk  cost  should 
be  budgeted  over  the  first  three  years  of  production  to  coincide  with 
the  production  rate  buildup  and  the  period  of  probable  disruption. 

PRACE  funds  should  be  available  when  needed  to  minimize  the  impact  of 
disruptions.  Figure  4.4  on  page  SB  illustrates  the  relationships  among 
the  BCE,  Total  Expected  Risk  Cost,  and  PRACE  for  an  example  PRAM  exercise. 

A  sensitivity  analysis  may  be  conducted  on  soft  data  to  determine 
impacts  of  any  estimating  errors.  Automating  the  PRAM  algorithm  facilitates 
sensitivity  analyses. 

A  PRACE  should  be  developed  at  least  three  times  before  Milestone  III. 

The  first  PRACE  should  be  prepared  roughly  two  years  prior  to  Milestone  Ill 
to  insure  funds  can  be  programmed  for  the  early  production  years.  A  PRACE 
can  be  developed  earlier  than  this,  say  at  ASARC  II,  but  there  is  still 


considerable  uncertainty  in  the  progr.im  then  tn.it  will  Pc  reduced 
later  in  FSED.  The  second  PRACE  should  he  prepared  about  a  year  after 
the  first  to  meet  t  tie  budget  regui  rement  s  ‘or  the  earlv  production  years. 
The  third  PRACF  should  be  prepared  to  supper*  the  ASARC  HI  decision. 

Each  update  will  be  more  accurate  than  the  previous  PRACF  as  additional 
information  is  available  and  t.nere  is  less  uncertainty  in  the  design. 

0 .  PRAM  EX E RCJ SE - 

The  following  example  illustrates  the  proposed  PRAM  and  establishment 
of  cost  distributions  for  the  various  risk  categories.  In  this  example, 
a  new  generation  weapon  system,  called  "Svst.-m-X , "  is  about  two  years 
prior  to  ASARC  111.  A  RCE  has  recent  1  /  beer  completed  ind  va'id.ited. 

A  PRACE  tor  the  first  three  years  of  production  is  to  *,r  derived  by 
investigating  each  risk  category. 

1 .  Management  Goa  1 s . 

This  weapon  system  is  viewed  by  Congres'  and  the  Defense 
establishment;  as  essential  to  nur  defence  over  ‘hr  re*t  rerade.  Its 
high  priority  and  nigh  price  teg  have  resuit-tv.  u  cons  idem:  I  e  pub]  ic 
scrutiny.  No  major  program  redirections  have  neon  dictated  bv  i:  thus 
far.  However,  due  to  increased  tension  in  the  Middle  East,  it  is 
anticipated  that  the  I  DC  will  be  moved  up  and  quantity  reu.ii cements  will 
be  increased.  Any  such  program  change  would  have  a  cost  impact  during 
the  transition  production  period  and  may  require  a  reprogramming  action 
with  recomputation  of  BCE  and  PRACE.  Therefore,  no  PRACE  cost  funds 
are  included  at  this  time. 

2.  Inflation. 

The  BCE  is  based  upon  a  DOD  directed  inflation  rate  of  6.S  per 


annum.  However,  this  is  only  an  estimate.  Economists  predict  that 
actual  inflation  during  this  period  could  run  as  low  as  4.0"'  or  as  high 
as  15.0  .  This  uncertainty  is  readily  translated  into  a  dollar  range 
around  the  BCE.  If  the  BCE  escalation  amount  (i.e  ,  difference  between 
constant  and  current  dollar  estimate  for  first  three  years  of  production) 
of  $200. M  is  treated  as  the  most  likely  value,  the  lower  limit  is  $140. M 
and  the  higher  limit  is  $660. M. 

3.  Unknowns . 

The  facility  planned  for  Igw  rate  initial  production  is  located 
along  an  earthquake  fault.  Minor  tremors  occur  infrequently  so  this  is 
viewed  as  low  risk.  However,  if  any  damage  does  occur  due  to  a  tremor, 
the  program  disruptions  may  eventually  impact  the  initial  production 
period.  Another  risk  is  that  labor  negotiations  at  the  prime  contractor 
are  due  approximately  6  months  prior  to  the  start  of  production.  A 
strike  is  viewed  as  having  a  low  probability  of  occurrence  and  a  cost, 
impact  cannot  be  probabi 1 i stical ly  attributed  at  this  point. 

4 .  Management . 

System-X  will  experience  unusually  high  management  risk  during 
initial  production  due  primarily  to  three  causes.  First,  the  PM0  does 
not  have  the  desired  level  of  production  experience  on  its  staff  and  is 
experiencing  difficulties  in  hiring  this  expertise  due  to  non-competitive 
salaries  with  private  industry.  Second,  the  prime  contractor  plans  to  he 
essentially  a  system  assembler,  relying  upon  an  unusually  large  number 
of  major  subcontractors  and  GFE.  Third,  the  relationship  between  the  PM0 
and  prime  contractor  is  becoming  increasingly  adversarial.  The  cost 
impact  of  potential  problems  in  this  area  of  management  were  not  incorporated 
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in  the  BCF.  Through  subjective  assessments  based  upon  analogous 
systems  and  discussions  with  MSC/PMO  personnel,  this  cost  impact  was 
estimated  as  equally  likely  between  5't  and  20'  of  the  BCE  amount. 

This  results  in  dollar  values  of  $45. M  to  $180. M,  uniformly  distributed. 

5.  Funds. 

The  prime  contractor's  cash  flow  position  is  currently  tight, 
with  funds  available  at  historically  high  interest  rates.  However, 
the  BCE  is  considered  to  be  Dased  on  reasonable  estimates  and  should 
cover  planned  activities.  So,  if  a  realistic  PRACE  is  approved  to  cover 
uncertain  activities,  funds  should  not  be  a  significant 
risk  category. 

6.  Facilities. 

This  is  considered  to  be  a  high  risk  category.  The  prime 
contractor  plans  to  build  a  new  production  facility  for  high  rate 
production  in  the  "Sun  Belt,"  far  from  the  current  n?<0  facility  which 
is  on  the  West  Coast.  Costs  to  build  and  equip  the  facility  were 
"guestimat.ed"  in  the  BCE,  but  there  are  a  number  of  uncertain 
activities  not  addressed  in  the  BCF.  At  this  early  date,  the 
design  and  production  aspects  have  not  stabilized  enough  to  determine 
actual  facility/equipment  requirements.  By  the  time  the  production 
specifications  have  stabilized,  there  may  not  be  enough  lead  time  left 
to  acquire  the  more  sophisticated  manufacturing  equipment  that  is 
anticipated.  Based  on  expert  assessment  of  anticipated  production 
requirements  and  equipment  lead  times,  the  cost  impact  over  and  above 
BCE  estimates  can  be  as  high  as  $20. M.  The  pM's  and  tne  prime  contractor' 
position  is  that  the  BCE  has  a  high  probability  of  achievement  with 


progressively  smaller  chance  oh  incurring  additional  costs  up  to  the 
projected  maximum. 

7.  Material . 

At  the  present  stage  in  SYSTEM-X's  life,  the  main  contributors 
to  risk  in  this  category  are  subcontracted  software  and  strategic 
materials.  To  meet  performance  requirements,  the  system  includes  numerous 
computers  and  integrating  software  and  three  separate  strategic  materials. 
Software  has  been  receiving  considerable  attention,  and  the  "buns"  are 
anticipated  to  be  solved  by  ASARC  III.  The  amount  included  in  the  BCE  for 
strategic  materials  is  considered  to  be  the  most  likely  value.  However, 
the  availability  and  price  uncertainty  of  the  strategic  materials  could 
result  in  additional  cost  impact  during  production.  On  the  pessimistic 
side, investment  speculation  in  such  "hard"  assets  could  conceivably 
drive  the  price  up  five-fold  which  translates  to  530. M.  On  the  optimistic 
side, this  same  commodity  market  could  bring  the  price  down  to  one  half 
of  the  price  estimated  in  the  BCE. 

8.  Labor. 

This  is  considered  to  be  a  high  risk  category  since  the  system 
will  be  produced  in  an  area  with  little  available  manufacturing  labor  and 
most  of  the  needed  labor  must  be  relocated.  Using  BCE  data,  it  is 
determined  that  $300. M  has  been  estimated  for  labor  cost  which  includes 
a  1 0"'  contingency  factor.  Labor  shortages  will  cause  disruptions  resulting 
in  increased  labor  expenses  of  overtime,  rework,  and  training,  for  example. 
An  additional  $180. M  is  determined  to  be  adequate  to  cover  the  increased 
cost  due  to  a  labor  shortage.  This  was  calculated  by  assuming  a  95' 
learning  curve  versus  the  88*  used  for  the  BCE.  This  results  in  a  labor 


cost  distribution  with  a  $270. M  low,  5300. M  from  the  BCE,  and  a  5480. M 
high  value. 


9.  Design  Stability. 

There  is  substantial  concurrency  built  intentional ly  into 
this  program.  The  rational^  is  to  correct  minor  design  deficiencies 
during  initial  production  and  hopefully  shorten  the  overall  program. 

The  intent  of  this  management  approach  is  to  reduce  the  long  term 
effects  of  inflation  and  reduce  the  total  program  cost.  The  risk 
involved  with  this  approach  is  that  design  instability  brings  uncertainty 
to  initial  production.  Therefore,  additional  funds  must  be  readily 
available  for  numerous  short  term  program  disruptions  stemming  from 
retests,  modifications,  ECP's,  workarounds,  stretchouts,  etc.  It  these 
funds  are  not  available  on  a  timely  basis,  the  disruptions  and  consequential 
cost  impacts  will  be  compounded.  A  Delphi  elicitation  provided  an 
estimated  cost  impact  of  S8.M  low,  $15. M  medium,  and  $40. M  high.  The 
BCE  for  related  activities  is  $10. M. 

1 0 .  Produci bi 1 i ty . 

The  R&D  program  has  a  small  amount  of  money  allocated  to  PEP. 

Up  to  this  point,  no  funds  have  been  expended  on  PEP  effort.  Based  on 
experience  with  analogous  but  less  sophisticated  systems,  the  System-X 
PEP  effort  should  be  more  substantial.  Assuming  that  all  the  PEP 
funds  will  be  spent  on  PEP  rather  than  being  diverted  to  solve  RXD 
crises,  there  will  still  be  risk  during  initial  production.  If  a  major 
producibil i ty  problem  occurs  during  initial  production,  the  cost  impact 
is  estimated  to  be  $12. M. 
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11.  Performance. 

Already,  several  changes  have  been  made  to  stringent  performance 
requi rements .  Strides  in  technology  coupled  with  a  changing  threat 
posture  have  resulted  in  several  PIP'S  and  more  are  anticipated.  The 
estimated  cost  impact,  including  probabi 1 istic  tasks  not  in  the  BCE  is 
equally  likely  to  occur  within  a  range  of  $15. M  to  $15. M. 

Summing  the  individual  risk  cost  amounts  for  this  example  yields  a 
Total  Expected  Risk  Cost  of  $352. 5. M.  Figure  4.3  shows  a  completed 
summary  for  the  example. 

The  total  risk  cost  must  also  be  allocated  over  the  first  three 
years  of  production.  Considering  such  things  as  the  desired  production 
rate  buildup  and  the  period  of  probable  disruption,  $145. M  is  planned 
for  the  first  year,  S 1 40. M  the  second,  and  $67. 5M  the  tnird  year. 

Figure  4.4  illustrates  the  breakout  and  relationships  among  the  BCE, 
expected  risk  cost,  and  PRACE  for  this  example. 

E.  SULTRY. 

PRAM  represents  more  than  just  a  breakout  of  risk  cost;  it  provides  a 
structure  to  describe  the  interrelationships  between  the  various  risk 
categories.  It  is  relatively  simple  to  apply  and  to  explain,  vet  is 
mathematically  rigorous  in  that  it  conforms  to  established  and  accepted 
statistical  techniques.  In  other  words,  it  is  useable. 

PRAM's  shortcomings  are  those  inherent  in  any  operations  research 
technique  that  requires  subjective  assessments  for  input.  Althounh  expert 
judgment  has  its  limitations,  it  should  not  be  discounted  as  being 
inappropriate  but  should  be  exploited. [33]  It  frequently  is  the  best 
approach  available. 
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TOTAL  EXPECTED  RISK  COST  SUMMARY 


TOTAL  ^XPFCFD  RISK  COST 


PRACE  BREAKOUT 


YEAR  1 

YEAR  2 

YEAR  3 

TOTAL 

INVESTMENT  PHASE  BCE. 

350. 

250. 

300. 

QOO.  0 

EXPECTED  RISK  COST 

145.0 

140.0 

67.5 

352.  r. 

PRACE 

495.0 

390.0 

367.5 

1252.5 

FIGURE  4.4  EXAMPLE  PRACE  BREAKOUT 
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CHAPTER  V 

CONCLUSIONS,  RECOMMENDATIONS,  AND  IMPLEMENTATION 
A.  CONCLUSIONS. 

The  Army  continues  to  encounter  problems  in  the  transition  of  a 
weapon  system  from  R&D  to  production  which  results  in  substantial  cost 
growth.  By  analyzing  11  Army  SAR  systems  during  this  transition,  it  was 
determined  that  a  35.5  f  average  cost  growth  was  occurring  (excluding 
inflation  and  quantity  changes)  in  the  Procurement  appropriation  which 
amounts  to  over  $5  billion.  Most  production  cost  growth  is  evidenced 
during  the  first  two  years  following  Milestone  III. 

As  in  R&D,  uncertainties  are  inherent  to  some  degree  in  all  new 
systems  entering  production.  Risks  cannot  be  totally  eliminated  in  the 
transition  of  an  Army  major  weapon  system  from  R&Q  to  production,  but  they 
can  be  reduced  through  such  means  as  good  Producibil ity  Engineering  and 
Planning  (PEP),  periodic  Production  Readiness  Reviews  ( PRR ) ,  and  experienced 
production  management.  Even  with  this  effort,  a  PMO  is  still  frequently 
faced  with  technological  uncertainties  carried  over  from  R&D,  and  he 
usually  is  pushing  the  state-of-the-art  in  manufactur i ng  processes  and 
techniques.  The  trend  toward  more  concurrency  will  increase  these  risks 
even  more. 

The  funding  environment  further  aggrevates  a  PMO's  oroblem  and  increases 
program  risks.  The  production  cost  estimates  must  be  made  early  during 
system  development  when  there  is  still  much  uncertainty  present.  Also, 
competition  among  and  within  the  Services  for  program  acceptance  forces 
optimistic  estimates.  These  optimistic  and  uncertain  estimates  are  then 
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established  as  targets  and  incorporated  into  the  budget.  Wnen  the  actual 
costs  are  known,  cost  growth  becomes  apparent  and  there  are  not  enough 
funds  available  to  meet  requirements.  The  program  then  experiences 
stretchout--which  disrupts  the  planned  production  system- -which  again 
can  increase  costs. 

Risk  analysis  techniques  were  analyzed  for  possible  use  in  assessing 
production  risk.  Methodologies  used  in  the  Army's  TRACE  program  were 
specifically  reviewed  as  they  are  generally  accepted  and  considered 
successful  in  addressing  the  R&D  area.  TRACE  has  reduced  significantly 
the  numbpr  of  funding  adjustments  required  for  Army  systems  in  FSED. 
Subsequently,  there  were  fewer  program  disruptions  due  to  funding  delays. 

It  should  be  pointed  out  that  almost  all  of  the  TRACF  deferral  funds 
available  to  PMO's  have  been  utilized  by  that  PMO.  This  is  contrary  to 
the  original  premise  on  which  TRACE  was  developed!  that  is,  roughly  half 
of  the  systems  will  experience  an  overrun  and  half  an  underrun.  It  is 
probable  that  this  will  continue  since  PMC's  will  always  nave  unfinanced 
requirements  to  justify  receiving  “their1  TRACE  funds  even  though  the 
funds  were  not  originally  determined  on  this  basis.  It  is  important  to 
realize  that  TRACE  relies  heavily  on  expert  judgment  to  estimate  the 
program  risks;  however,  by  its  very  nature,  program  risk  assessment  is 
a  subjective  process.  Finally,  even  though  TRACE  has  helped  manage 
R&D  cost,  it  has  not  eliminated  cost  growth. 

The  TRACE  techniques  currently  used  to  develop  R&D  estimates  can  be 
used  to  develop  production  estimates  as  well.  There  are  many  similarities 
in  the  two  areas.  The  uncertainties  are  just  of  a  different  nature. 

But  a  different  perspective  of  initial  production  uncertainty  was  taken 
to  improve  upon  the  techniques. 
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A  recognition  that  past  production  problems  represent  future  production 
risk  areas  suggested  an  empirically  developed  risk  structure  which 
specifically  addresses  initial  production  risk.  The  risk  structure  evolved 
such  that  it  captured  and  isolated  fundamental  production  uncertainties, 
maintained  as  much  independence  as  practical  among  toe  individual  ns* 
categories,  and  captured  the  synergism  of  the  BCE  investment  cos'  elements. 
The  Production  Risk  Assessing  Methodology  (PRAM)  was  developed  by  combining 
this  risk  structure  with  conventional  cost  estimating  techniques  to 
quantify  initial  production  risks  into  a  dollar  estimate  called  Production 
Risk  Assessing  Cost  Estimate  (PRACE).  The  PRACE  complements  and  improves 
upon  the  BCE  by  including  consideration  for  those  prnoucfion  risks  not 
addressed  in  the  BCE. 

The  PRACE  concent  can  reduce  production  risk  and  cost  growth. 

Use  of  PRACE  funds  will  helo  reduce  the  number  of  oronram 
disruptions  caused  by  funding  shortages  just  as  TRACE  has  none  for  RAD. 

It  will  also  assist  the  PMO  in  planning  for  the  transition  and  providing 
more  realistic  production  cost  estimates.  TRACE,  in  conjunction  w’th  on¬ 
going  Army  initiatives,  will  improve  the  acquisition  process. 

B .  RECOMMENDAT IONS . 

The  PRAM  has  the  potential  to  assist  in  analyzing  initial  production 
risk  and  incorporating  risk  funds  into  the  BCE.  Accordingly,  the 
following  recommendations  are  made: 

1.  The  PRAM  should  be  evaluated  by  being  applied  to  those  Army  systems 
recently  entering  production.  The  appropriate  PMO's  should  be  requested 
to  exercise  the  PRAM  by  reconstruction  their  programs  to  the  time 
period  roughly  two  years  prior  to  their  Milestone  III.  The  PRACf's 
produced  by  the  exercises  should  then  be  compared  to  the  current  system 
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cost  status  tor  comparability.  These  exercises  will  serve  to  test  arid 
evaluate  the  methodology  but  primarily  will  provide  a  data  base  of  recent 
experience  with  production  problems  within  the  PRAM  framework  This  data 
base  will  be  useful  in  preparing  future  PPACE's.  (bote  Imnlpmeotat ion : 
this  is  being  accomplished). 

2.  PRAM  should  also  be  exercised  by  a  HQ  DARC.DM  cost  team  on  a 
system(s)  other  than  the  above  SAR  systems  to  provide  an  independent 
acceptance  test.  The  test  system! s )  must  have  adequate  data  to  reconstruct 
the  time  period  from  rouqhlv  two  years  prior  to  Milestone  III  through  at 
least  three  years  of  production  (More  Implementation ;  this  is  complete!. 

i.  Assuming  PRAM  acceptance,  it  should  at  niipleim-'ced  wit  Mr  cm 
Army  through  the  issuance  of  a  Letter  of  Instruction  ('(.PI)  similar  to  the 
TRACE  LOT.  The  following  implementation  guidelines  are  su^  jeMcd. 

a.  A  PRACE  should  be  developed  at  least  tnree  times  during  the 
transition  period  using  the  PRAM.  The  first  exercise  should  tak^  place 
roughly  two  years  prior  to  Milestone  ill  to  insure  tne  PRACE  ran  be 
programmed  for  the  early  production  years.  Likewise,  the  second  exercise, 
which  becomes  an  update  of  the  first,  should  take  place  a  vear  or  so 
before  Milestone  III  to  meet  budget  requirements.  The  third  PRAM  exercise 
should  be  part  of  the  submission  to  ASARC/DSARC  III  to  assisi.  in  their 
production  decision. 

b.  The  PRACE  should  be  presented  as  part  of  the  Investment  Cost 
documentation  for  the  BCE  in  cost  element  2.11,  "OTHER."  The  PRAM  Summary 
and  supporting  rationale  and  calculations  should  be  available  for  review. 

c.  The  PRAM  exercise  should  be  a  team  effort,  independent  from 
the  PMO  with  at  least  three  team  members.  Suggested  team  members  are  one 
cost  analyst  from  HQ  DARCOM  and  one  production  engineer  and  one  cost 
analyst  from  the  appropriate  MSC .  PMO  personnel  should  be  relied  upon  to 
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providi  in  fonii.it  ion  and  to  ir,',!  j;  witm  e  needed  in  the  analysis, 
be  feasible  to  align  the  PRAM  analysis  with  t fie  PRR's,  <■  .ill  are 
concerned  with  reduction  risk  and  cu-a  . 

d.  Tne  PRACE  should  be  administered  within  t!r-  Army  just  as 
TRACE  for  R&D  currently  is.  funds  approved  tor  the  PRACE  should  be  held 
by  ODCSRDA  until  justified  "'or  release  by  a  PMO.  Over  time,  a  dat.i  oa s e 
of  PRACE  justifications  and  experience  will  develop  that  ran  be  used  to 
determine  and  refine  the  PRACE1 s  for  future  systems. 

e.  PRACE  submissions  should  be  mandatory  for  Army  major  weapon 
systems.  Since  it  has  been  shown  that  most  initial  production  problems 
and  cost  growth  have  surfaced  within  the  first  three  years  of  production, 
PRACE  funds  should  be  available  primarily  for  the  first  three  years  of 
production  with  some  exceptions  avowed  where  there  is  a  low  rate  initial 
production. 

4.  PRACE  concepts  and  methodologies  should  be  added  ?o  the  curriculum 
of  appropriate  courses  at  the  Army  Logistics  Management  Center  and  the 
Defense  Spstems  Management  College.  A  short,  intensive  seminar  should  be 
developed  to  train  those  analysts  who  will  or  may  develop  a  PRACE. 

5.  A  follow-on  study  should  be  initiated  to  develop  early  indicators 
of  production  cost  growth  that  can  be  used  to  improve  the  PRACF.  Some 
specific  areas  that  appear  to  have  nood  potential  are  a  follows: 

a.  The  relationship  of  cost  growth  to  PEP  funds  as  a  percentage 
of  RDT&E  funds  should  be  investigated. 

b.  The  relationship  of  long  leadtime  item  experience  , including 
initial  production  facilities)  to  production  cost  growth  mav  prove  useful 
as  an  early  indicator. 
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l.  Staffing  patterns  within  PMO  regarding  production  engineering  find 
management  may  explain  a  portion  of  the  cost  growth. 

d.  The  relationship  of  major  design  problems  to  production  problems 

in  that  design  area  may  provide  an  early  indicator  of  potential  cost  growth. 

e.  Finally,  the  pattern  and  amount  of  cost  arowth  experienced  by  a 
system  during  FSED  may  be  a  useful  indicator  of  production  cost  growth. 

(Note  Implementation;  this  is  complete). 

C .  IMPLEMENTATION. 

Between  the  time  the  draft  report  was  prepared  and  the  final  report 
was  published,  recommendations  2  and  5  were  implemented  and  implementation 
began  on  recommendation  1.  Also,  the  PRAM  was  incorpora  ted ,  with  some 
modifications,  into  an  approved  DARCOM  methodology  titled  Total  Risk 
Assessment  Cost  Estimating  for  Production  (TRACE/P).  TRACE/0  is  beinq 
applied  where  appropriate  to  develoo  cost  estimates  which  include  con¬ 
sideration  for  initial  production  risk. 
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